SQuARM-SGD: Communication-Efficient Momentum SGD for Decentralized Optimization

In this paper, we propose and analyze SQuARM-SGD, a communication-efficient algorithm for decentralized training of large-scale machine learning models over network. each node performs fixed number local SGD steps using Nesterov's momentum then sends sparsified quantized updates to its neighbors regulated by locally computable triggering criterion. We provide convergence guarantees our general (non-convex) convex smooth objectives, which, the best knowledge, is first theoretical analysis compressed with updates. show that rate SQuARM-SGD matches vanilla SGD. empirically including in can lead better test performance than current state-of-the-art which does not consider